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From tacit sensorimotor coupling to articulated mathematical reasoning in an 
embodied design for proportional reasoning^ 

Dor Abrahamson, Jose F. Gutierrez, Rosa G. Lee, Daniel Reinholz, & Dragan Trninic 
Embodied Design Research Laboratory, Graduate School of Education, UC Berkeley 

Mathematies edueation designers and researehers are beginning to 
appreciate the pedagogical potential of embodied interaction (El) 
instructional activities, yet little theory is available to understand its 
historical roots, sociocognitive mechanisms, and implementation practice. 

We draw on empirical data {n = 22) from a Grades 4-6 El interview-based 
exploratory design for proportionality that involved “What’s-my-rule?” 
remote-controlling of virtual objects on a computer display. Students were 
guided to reinvent mathematical concepts by reflectively instrumenting 
entrained perceptuomotor competence with interpolated symbolical 
artifacts. Analyzing qualitatively for critical dimensions, we propose and 
demonstrate three interaction milestones enabling or constraining learners’ 
participation in conceptual El activities: (a) re-seeing particular design 
features as peripheral-technological rather than central-mathematical; (b) 
re-seeing some would-be pragmatic actions as epistemic; and (c) re-seeing 
symbolic artifacts as affording or enhancing the enactment, explanation, or 
evaluation of hands-in immersive solution procedures. We delineate 
emergent principles for an effective El mathematics design framework. 



1. Introduction 

LI What is Embodied Interaction? 

Embodied interaction (El) is a form of technology-supported training activity created, 
implemented, and researched by scholars interested in investigating multimodal learning. 
Through engaging in El activities, users build schematic perceptuomotor structures 
consisting of mental connections between, on the one hand, physical actions they perform 
as they attempt to solve problems or respond to cues and, on the other hand, automated 
sensory feedback on these actions. One objective of El design is for users to develop or 
enhance targeted schemes that undergird specialized forms of human practice, such as 
mathematical reasoning. As is true of all simulation-based training, El is particularly 
powerful when everyday authentic opportunities to develop the targeted schemes are too 
infrequent, complex, expensive, or risky. Emblematic of El activities, and what 
distinguishes El from “hands on” educational activities in general, whether involving 
concrete or virtual objects, is that El users’ physical actions are intrinsic, and not just 
logistically instrumental, to obtaining information (cf. Marshall, Cheng, & Euckin, 2010). 
That is, the learner is to some degree physically immersed in the microworld, so that 
linger, limb, torso, or even whole-body movements are not only in the service of acting 
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upon objects but rather the motions themselves become part of the perceptuomotor 
structures learned. If to use David Kirsch’s taxonomy, all El gestures constitute epistemic 
actions, even if they are initiated as pragmatic actions (see Kirsch, 2006). In El, the 
learner’s body becomes concrete instructional material. El is “hands m.” 

Typically, El activities emphasize explorative perceptuomotor tasks and draw less on 
propositional or domain-specific reasoning (e.g.. Antic, Corness, & Droumeva, 2009). 
Notwithstanding, El activities may include standard symbolic elements, such as 
alphanumeric notation, diagrams, and graphs (Cress, Eischer, Moeller, Sauter, & Nuerk, 
2010; Nemirovsky, Tierney, & Wright, 1998). Indeed, content-oriented El activities are 
often designed explicitly to foster the guided emergence of domain-specific conceptual 
structures from domain-neutral perceptuomotor schemes. 

1.2 Why Should Learning Scientists be Interested in Embodied Interaction? 

This paper discusses documented evidence of students who struggle with core aspects of 
subject matter content through engaging in El -based instructional activities. Such studies 
should be of particular interest to scholars who are developing models of learning 
informed by both cognitive and sociocultural theory (the “dialectical” approach, see 
diSessa, 2008), because the studies generate interaction data that bring out in relief 
mediation practices for the cultural signification of discipline-neutral schematic action. In 
particular, we are interested in theorizing how social interactions steer learners to 
leverage perceptuomotor competence in the appropriation of disciplinary forms of 
reasoning. Because our research is cast specifically within educational contexts of 
practice, including artifact design and actual teaching, we ask: 

■ What are effective educational practices enabling students to develop and then 
draw on perceptuomotor competence as a resource for conceptual learning? 
Specifically: 

o What are productive El design principles? Given that computational 

environments enable designers to couple literally any physical action with 
any sensory feedback by interpolating algorithmic links between them 
(even if this coupling is ecologically atypical), which specific coupling 
would support the learning of some particular targeted concept? 

o What are the instructor’s roles in orchestrating individual learners’ 
development of perceptuomotor skill and then signifying it with 
mathematical meanings? In particular, what, if any, are the interaction 
challenges an instructor may encounter in supporting learners’ articulation 
of their perceptuomotor skill? On the other hand, what, if any, are 
unexpected pedagogical affordances inherent to El? 

1.2 What Are Our Broader Research Objectives? 

At its broadest, then, as learning scientists we are concerned with the general research 
problem of conceptual ontogenesis — how individuals develop higher-order ideas. The 
theoretical problem of conceptual ontogenesis is worthy of scholarly pursuit even just in 
terms of its critical implications for educational policy. In particular, this research 
consistently fuels and is fueled by heated policy debates over the controversial use of 
concrete objects and imagery in mathematics curriculum (e.g.. Brown, McNeil, & 
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Glenberg, 2009; Schoenfeld, 2004; Sloutsky, Kaminski, & Heckler, 2005). In turn, this 
problem is informed by a spectrum of perspectives on cognitive and cultural sources of 
concept development (e.g., Carey & Gelman, 1991; cf Newman, Griffin, & Cole, 1989). 

We view our research paradigm as potentially illuminating of these theoretical and 
entailing policy debates over conceptual ontogenesis and the roles of media therein. 
Namely, prior research efforts stemmed from, and are positioned in distinct disciplinary 
traditions: they draw on diverse and even disparate epistemological assumptions with 
respect to the nature of teaching and learning and consequently use different, even 
incompatible methodology to pursue their intellectual or ideological agenda. As a result, 
we maintain, prior attempts to offer a comprehensive view of conceptual ontogenesis 
have by and large been encumbered or even precluded by partisan discourses that have 
rarely offered routes of reconciliation (see also Artigue, Cerulli, Haspekian, & Maracci, 
2009, on the challenges of dialogue among diverse learning-sciences theoretical models 
of learning). Specifically, few studies on mathematical ontogenesis are “dialectical” 
(diSessa, 2008), in the sense that they attempt to model the nuanced and tight-knit 
reciprocity of cognitive agency in the social mediation process (but see, e.g., Sfard, 

2002); few studies are multimodal, in the sense that they attend beyond verbal utterance 
also to action and gesture (but see, e.g., in Edwards, Radford, & Arzarello, 2009; Hall, 
1996); so that fewer studies yet are both dialectical and multimodal in the sense that they 
offer models of how individual learners participating in cognitively demanding organized 
social activity build and sustain common-sense meanings for properties of objects they 
engage and scrutinize, even as they embody, inhabit, and signify mathematical practices 
grounded in these situated experiences (but see, e.g., Roth, 2009). 

1.4 What Are the Methodological Affordances of El for Research on Learning? 

Arguably, El creates powerful methodology for dialectical investigations of conceptual 
ontogenesis. To begin with, by augmenting the learning environment with computational 
algorithms and virtual objects, designers can engineer highly specified perceptuomotor 
coupling, such as coupling a precisely executed coordinated bimanual gesture with 
“correct’ ’/“incorrect” color feedback, that are customized to support the instruction of 
hitherto challenging mathematical concepts in the spirit of reform-oriented pedagogical 
frameworks yet would be impossible or too awkward to implement using pre-electronic 
media. Thus, researchers enjoy opportunities to evaluate the merit of certain pedagogical 
frameworks with less implementation “noise.” Moreover, recent technological advances 
in remote action now enable net perceptuomotor training, that is, “au naturel” interaction 
unencumbered by mediating instruments or interfaces whose acquired online operation 
taxes conceptually irrelevant cognitive resources. El thus affords better opportunities to 
evaluate the potential of an approach to the design of mathematics learning materials that 
attempts to ground conceptual ontogenesis in onsite developed perceptuomotor 
competence. Einally, the very specific perceptuomotor schemes that El activities foster 
are probably peculiar to those settings — ^we can quite safely trace their origin to our own 
design — thus lending greater validity to claims regarding the effect of particular 
interventions on learning outcomes. Eor example, any valuable mathematical insight that 
students manifest in the pedagogical context of stacking wood blocks could not be traced 
with confidence to that design alone, because children stack objects in regular extramural 
activities; yet these children rarely move their hands at different yet coordinated 
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velocities, so that a design supporting this new skill could be implicated as its probable 
source. The singularity of El design may thus enable a comparatively uncontaminated 
tracking of ontogenesis. 

1.5 What Are Our Specific Research Objectives? 

Our objective in this paper is to offer some tentative dialectical reflections on conceptual 
ontogenesis — ^what it is grounded in and how it can be fostered. These reflections draw 
on a recent study, in which we investigated an instructional methodology for scaffolding 
the emergence of proportional reasoning from El problem-solving activities of our design 
(Howison, Trninic, Reinholz, & Abrahamson, in press; Reinholz, Trninic, Howison, & 
Abrahamson, 2010). As reflective designers who believe that constructs, insights, and 
heuristics should guide, emerge from, and apply beyond the particularity of specific 
artifacts (diSessa & Cobb, 2004; Schon, 1983; Schunn, 2008), we hope with this work 
also to contribute more generally to the theory and practice of El-based mathematics 
instruction, which we view as bearing great promise. Essentially what we are trying to 
accomplish in this exploratory study is to parameterize El, which is still a fledgling 
practice with little educational exemplars or theory, toward building a comprehensive 
framework for the design, facilitation, and evaluation of pedagogical, concept-oriented El 
activities. As such, we view as potentially useful to the field any coherent theoretical 
constructs and models we may offer that lend insight into how El works, such as our 
characterization of pedagogical El as first nurturing new perceptuomotor competence and 
then progressively cultivating it into conceptual structures. 

The paper is structured as follows: after we lay out our claims, discuss EEs diverse 
origins, and explain the particular design we have been developing and researching, we 
present several empirical episodes from our early usability testing that we view as 
particularly illuminating of educational challenges and opportunities at the 
perceptuomotor-conceptual interface. 

1.6 What Are the General Claims of This Paper? 

Building on analyses of our empirical data, we argue that El bears promise for the theory 
and practice of mathematics education. El design, we contend, can create sites for 
presymbolic struggle with, and resolution of core conceptual challenges of pedagogically 
targeted subject matter content. Though these presymbolic perceptuomotor struggles are 
not “conceptual” in the classical sense of explicit association with particular disciplinary 
practice, including vocabulary, procedures, inscriptions, and broader institutional 
enframings, we view these struggles as critical to effective participation in these 
disciplinary activities. Our claim draws on empirically grounded theoretical views that 
the cognitive resources of mathematical reasoning are not symbolic propositions but 
presymbolic dynamical images (Barsalou, 2010; Goldin, 1987; Eakoff & Nunez, 2000; 
Pirie & Kieren, 1994; Presmeg, 2006), and we are further inspired by calls to ground 
mathematical learning in everyday intuition (Eorman, 1988; Kamii & DeClark, 1985; 
Resnick, 1992; Thompson, 1994). 

What form do these alleged presymbolic struggles take? We looked closely at our video 
footage of El study participants as they were developing the design’s targeted physical 
enactment. As we will elaborate in the empirical section of this paper, El users develop 
new perceptuomotor competence by reinterpreting the planning and executing elements 
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of their interaction. In particular, users who initially enact a naive theory-in-action for 
completing the task objective may interpret the automated “error” feedback they receive 
as indicating not an incorrect theory but rather execution imprecision. We use the terms 
“theory” or “theorem-in-action” to depict contextual problem-solving principles learners 
heuristically determine and subsequently entertain and evaluate as they investigate 
situations they attempt to control (Karmiloff-Smith, 1988; Karmiloff-Smith & Inhelder, 
1978; Pufall, 1988; Vergnaud, 1983, 2009). Thus students’ theory, which stems from 
their prior intuitions, is a double-edged sword (Cobb, 1989). That is, designers hope 
students will develop new theory in light of patterns in the automated feedback on their 
perceptuomotor enactment, yet students might be oblivious to these patterns and 
perseverate in confirmation-bias interaction. How then might students both draw on and 
qualify or modify their naive theory? 

We look at the roles of discursive and mathematical resources in guiding learners to re- 
see their own perceptuomotor enactment in pedagogically productive ways. Our analyses 
implicate the vital role of mathematical symbolic structures (e.g., a Cartesian grid) and 
inscriptions (e.g., numerals) available in a problem-solving space as the means by which 
students “re-describe” (Karmiloff-Smith, 1992) their entrained perceptuomotor enactment 
in appropriate disciplinary form. We demonstrate how these redescriptions can take 
surprising, pedagogically productive directions as students discover contextually 
advantageous ways of using the symbolic artifacts to enact, explain, or evaluate their task 
strategy. As such, we expand on neo-Vygotskian conceptualizations of appropriation 
(e.g., Bartolini Bussi & Mariotti, 2008; Saxe, 2004; Sfard, 2002) by implicating learners’ 
creative reappropriation of symbolic artifacts in ways that are not mediated by the 
instructor yet are discovered in the discursive context as the artifacts’ task-specific 
emergent semiotic-enactive affordances (cf. Hall, 2001; Neuman, 2001). 

More generally, we submit, El activities constitute rewarding empirical contexts for 
research aiming to deepen an understanding of how instructors discipline learners’ 
perception of a shared domain of scrutiny (Goodwin, 1994; Hall, 1996; Stevens & Hall, 
1998). Interestingly, the objects-to-think-with in our El domain of scrutiny are not static 
and substantive but physical theory-in-action (Abrahamson & Howison, 2010). 

1 . 7 Stepping Back to Move Forward: Recent Flistorical Roots of Embodied Interaction 

The historical roots of El in science and philosophy are numerous and varied. As a form 
of instructional activity, El design emulates genetic epistemology (Piaget, 1968) and, in 
particular, its thesis that conceptual change is mediated by goal-oriented sensorimotor 
interaction followed by reflective abstraction on these interactions (for a converging view 
from "Constructionism," see also Papert, 1980, on "body syntonicity"). El rationales can 
be further affiliated with broad principles of Phenomenology implicating immersed 
action as epistemically antecedent to reflective reasoning (e.g., Husserl, 1970). El design 
is often inspired by grounded-cognition research and, notably, the empirically supported 
conjecture that human reasoning transpires not as processing amodal symbolically 
encoded propositions but as simulating fragments of previously experienced modal 
activity (Barsalou, 1999, 2010; Dourish, 2001; Hommel, Mrisseler, Aschersleben, & 
Prinz, 2001; Varela, Thompson, & Rosch, 1991). That is, as some cognitive linguists 
have long been arguing, human reasoning — including mathematical reasoning — utilizes 
kinesthetic image schemas from everyday mundane activity that are projected as 
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conceptual metaphor to generate and eommunieate thought (Johnson, 1987; Lakoff & 
Nunez, 2000; Nunez, Edwards, & Matos, 1999). Curiously, the aetion-before-eoneept 
tenet from philosophy and eognitive seienee resonates well with soeioeultural aetivity 
theories of knowledge mediation (Engestrom, 1999; Wertseh, 1979) as well as with 
eultural-historieal precedents of indigenous pedagogieal praetice (e.g., Kammer, 1986). 
Einally, beeause El often uses remote aetion, a modality of partieular interest to El 
praetiees and researeh is manual gesture — hand movements that do not aet upon aetually 
present substantive or virtual objeets but rather are assoeiated with eommunication and 
reasoning, sueh as about aeting upon non-present imaged objects. Eaboratory studies 
repeatedly demonstrate the mediating role of gesture in the generation, teaehing, and 
learning of diseiplinary eoneepts and proeedures; gesture has the eapaeity to evoke 
multimodal images, coneretize emerging notions, implieitly eommunieate information 
and forms of reasoning, and frame pereeption (for a reeent review, see Goldin-Meadow 
& Beiloek, 2010; see also Yap, So, Melvin Yap, Tan, & Teoh, 2011). Gesture is thus 
relevant to analyzing diseourse around El, when partieipants mime their aetions and 
tutors refer to these gestural performances. 

In sum, the manifold roots of El in seholarly exposition have eonverged and grown to a 
tipping point where, eoupled with reeent teehnologieal advanees, El stands to become a 
foeus of design for, and researeh on mathematical learning. As design-based researchers 
of mathematieal eognition and instruetion, we eonsider El aetivities useful empirieal 
settings for researeh on the ontogenesis of mathematieal eoneepts. These immersive 
aetivities ereate tension between unreflective orientation in a multimodal instrumented 
spaee and refleetive mastery over the symbol-based redeseription of this aequired 
competenee (Nemirovsky, Tierney, & Wright, 1998). In partieular, our researeh is 
eentered on understanding how teohnologieally instrumented proeedural performanee 
(Dreyfus & Dreyfus, 1999; Klemmer, Hartmann, & Takayama, 2006) gives rise to the 
enactment of soeioeognitively normative mathematieal praetiee ("mathematical signs" 
grounded in "artifaets signs," see Bartolini Bussi & Mariotti, 2008; "situated 
abstraetions," seeNoss, Healy, & Hoyles, 1997). 



2. Design 

Our design was propelled by a eonjecture that El aetivities might supplement everyday 
mundane aetivity technologieally by ereating opportunities for learners to develop image 
sehemas that are vital to mathematieal learning yet hard to eome by outside of dedieated 
instruetional eontexts. Namely, we sought to build an interaetive eomputational system 
supporting the entraining of proportional transformation as well as a set of mathematical 
instruments for students to rearticulate their emerging strategies in ways that would be 
useful for their understanding of, and flueney in proportional and related construets. 

Speeifieally, we conjeetured that students’ eanonieally ineorrect solutions for rational- 
number problems — “fixed differenee” solutions (e.g., "2/3 = 4/5" - Eamon, 2007) — 
indieate students’ laek of multimodal action images to ground proportion-related 
eoneepts (ef. Pirie & Kieren, 1994). We thus built a mieroworld wherein the physieal 
solution operations inseribe the kinesthetie image schema of the emerging mathematieal 
notions. Study partieipants were teehnologieally steered to move their arms in a 
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choreographed form that would subsequently emerge through mediation as proportional. 
A sequenee of mathematical instruments interpolated into the microworld supported the 
progressive mathematization of these actions (that is, their signification in formal 
mathematical inscriptions). 

At the center of our instructional design is the Mathematical Imagery Trainer (hence 
"MIT" - see Figure I and Figure 2, below. For detailed deseriptions of the deviee's 
rationale and teehnieal properties as well as initial empirical results, see Reinholz et ah, 
2010; Trninie, Reinholz, Howison, & Abrahamson, 2010). 







a. 



b. 





d. 



Figure 1. MIT interaetion sehematies, with the device set at a 1:2 ratio, so that the right 
hand needs to be twice as high than the left hand; (a) incorrect performance (red feedback 
on exploratory gestures); (b) almost correct performance (yellow feedback); (c) correct 
performance (green feedbaek); and (d) another correet performance. 




a. b. 

Figure 2. MIT in action: (a) “incorrect” enactment turns the screen red; and (b) “correct” 
enactment turns the screen green. For a 5 minute video clip showing the MIT in action, 
see http://edrl.berkelev.edu/eontent/mathematics-imagerv-trainer-video-pmena32 . 

The MIT measures the height of the users’ hands above the desk. When these heights 
(e.g., 10” & 20”) mateh the unknown ratio set on the interviewer’s console (e.g., 1:2), 
the sereen is green. So if the user then raises her hands in front of this presymbolic 
“What’s-my-rule?” artifact in accord with the ratio (e.g., to 15” & 30”), the screen will 
remain green but will otherwise turn red (e.g., to 15” & 25”, a coordinated hand 
elevation that maintains fixed distanee of 10”). Study participants were tasked first to 
find green then to maintain it while moving their hands. The protoeol included layering a 
set of mathematical artifacts onto the display, such as an adaptable Cartesian grid (see 
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Figure 3, below), to eatalyze progressive mathematization of emergent strategies. 

Partieipants ineluded 22 students from a private K-8 suburban sehool in the greater Bay 
Area (33% on finaneial aid; 10% minority students). Care was taken roughly to balanee 
students both by gender and by low-, middle-, and high-aehieving groups as ranked by 
their teaehers. Students partieipated either individually or paired in a semi-struetured 
elinieal interview (duration mean 70 min.; SD 20 min.). Interviews eonsisted primarily of 
working with the MIT. At first, the condition for green was set at a 1 :2 ratio, and no 
feedback other than background color was given (see Figure 3 a, below; we used this 
challenging initial condition only in the last six interviews). Then, crosshairs were 
introduced (see Figure 3b); these virtual objects mirrored the location participants’ hands 
in the interaction space yet, so doing, became the objects users acted on, then through. 
Next, a grid was overlain on the display (see Figure 3c) to help students plan, execute, 
and interpret their manipulations and, so doing, begin to articulate quantitative verbal 
assertions. In time, numerical labels “1, 2, 3,...” were overlain along the grid’sy-axis 
(see Figure 3d): these enabled students to construct further meanings by more readily 
recruiting arithmetic knowledge and skills and distributing the problem-solving task. 
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b. 



c. 



d. 



Figure 3. Display configuration sketches: (a) continuous-space mode; (b) continuous- 
space mode with crosshairs, i.e. the virtual objects that users manipulate; (c) crosshairs 
with grid overlay; (d) crosshairs with grid overlay and y-axis numerals. 



Having briefly described our design and data-gathering methods, we now turn to discuss 
several features of conceptual El activities that we discovered through collaborative 
microgenetic analysis of our data corpus (Schoenfeld, Smith, & Arcavi, 1991). In accord 
with elements of grounded theory (Glaser & Strauss, 1967), when we encountered in our 
video footage interaction behaviors of interest to our existing and emerging research 
questions, we sought to determine whether these behaviors were consistent within and 
between students in our larger data corpus. As a result, we winnowed out rare, though 
intriguing behaviors yet identified general cross-participant patterns. Nevertheless, our 
report on these patterns is located back in individual cases so as to lend authenticity to 
these patterns as they manifest in idiosyncratic contexts of inquiry and interaction. 
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3. Features of Conceptual El Activities — an Empirical Account 

This study has been our first attempt at designing for El mathematical learning. The 
objective of this chapter is to describe, analyze, and generalize several features of our 
empirical data that we see as potentially paradigmatic of using these new media, with the 
hope that these narrative accounts, analyses, and constructs will prove helpful for other 
researchers working with these tools. Section 1 describes participants’ struggles to reach 
entry-level engagement with our El technology. Section 2 discusses mediated passages 
from tight perceptuomotor coupling with the technology to global proto-proportional 
schemes, and Section 3 relates students’ creative reappropriation of mathematical 
artifacts in ways that extend their demonstrated use in pedagogically desirable directions. 

5 . 1 From Peripheral to Mathematical Interaction 

When instructional designers create artifacts, students may engage them in unexpected 
ways that lead them astray of the designers’ intended interaction trajectory. This is true of 
innovative electronic artifacts in particular (Olive, 2000; White, 2008), possibly because 
these artifacts are historically new and so both designers and users are still developing an 
understanding of their affordances. Eor example, users of augmented-reality technology 
who manipulate tactile artifacts linked to virtual simulacra tacitly expect the virtual 
objects to interact in accord with their simulated physical ontology, yet those affordances 
may not have been engineered into the computational space (Hornecker & Diinser, 2009). 

Users’ unintended interactions with novel artifacts present nuanced issues for design- 
based researchers. On the one hand, a major concern of instructional designers is to 
gather empirical information on how prospective users interact with prototype products 
under cycles of iterative development. Moreover, design-based researchers are explicitly 
interested in users’ idiosyncratic engagement with their artifacts, because the researchers 
view these diverse forms of engagement as hinting to the cognitive structures they wish 
to investigate and cater to (Abrahamson, 2009). Yet, on the other hand, design-based 
researchers are liable to focus their work on interactions that take place in their media of 
choice and make implicit assumptions that their users are able to meet prerequisite 
criteria of entering these media. By way of analogy, paperback writers assume that 
prospective consumers of their products can read and, a fortiori, that they are able to open 
a book and turn pages. Yet whereas designers distinguish clearly between para-media and 
in-media aspects of engaging their artifacts, and in fact often wish to make the para- 
media components “invisible” (Raymaekers, 2009, p. 2), this distinction may be entirely 
lost on users who are new to a medium, such as El. New to a medium, users are 
unequipped to determine whether their failure to achieve task objectives is due to para- 
medium peripheral issues of operational fluency or in-medium substantive issues of 
inquiry heuristics, strategy, or content knowledge. Moreover, when artifacts under 
development are still “buggy” or “klunky,” users can hardly be expected to navigate the 
phenomenological amalgam of designers’ unintended and intended task features. 

Arguably, a paper on mathematical learning need not trouble the reader with para-media 
interaction foibles and instead should highlight in the transcriptions only incidents of 
conceptually relevant interactions. And yet, given that El activities are still quite new, we 
see value in sharing the user-experience (UX) consequences of our yet-imperfect 
technology. Thus, in this section, we compare two cases of interviewees who were either 
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successful (Asa and Kaylen) or not too successful (Boaz) in entering our El medium, and 
we speeulate on dimensions of interaetion relevant to understanding this differenee. In 
both oases, the tutors were graduate-student researohers, oo-authors of this paper who 
were apprentioing as olinioal interviewers using a semi-struotured protoool, and the PI 
was present to guide and oooasionally intervene. 

We begin with Asa and Kaylen, 5*-grade male students indioated by their teaohers as low 
aohieving, and foous on Asa’s reasoning at the introduotion of the orosshairs. In this 
dyad-interaotion task, the students eaoh hold and manipulate one of the two traoker 
devioes and attempt to oo-produoe a green soreen. Prior to the layering of the orosshairs 
into their interaetion spaoe, the ohildren were observed to alternate their peroeptual 
attention baok and forth between the hand-held devioes and the oomputer display. They 
looked at their devioes so as to note their relative positions (e.g., to maintain or ohange 
the distanoe between them), and they looked at the soreen so as to note the oolor feedbaok 
(whether it were green, yellow, or red). Even when lifting their hands to the level of their 
eyes, suoh that the devioes were in the same line of vision as the soreen, the partioipants 
oontinued to adjust their gaze baok and forth. The following transoript oaptures Asa’s 
utteranoe prior to the introduotion of the orosshairs. 

Asa: Oh 1 see. So 1 think what’s going on is that they [referring to two hand- 

held devioes] have to be the same far. . . [RH thumb and index gestures a 
vertioal interval] the same distanoe away from eaoh other [RH, still 
holding the interval, gestures toward Kaylen’ s devioe]. 

Asa thus offered what we refer to as a “fixed distanoe” manipulation oonjeoture, namely 
he believed that in order to move the handheld devioes yet maintain a green soreen, the 
interval between the hands should be invariant (see below Asa’s “theory”). We also note 
Asa’s use of the pronoun “it” in “keep it the same distanoe” that suggests Asa has 
objeotified the vertioal interval between the two devioes as the thing to be oontrolled. 

JEG reveals the orosshairs on the oomputer display 

JEG: Can you see that? 

A/K: Yeah. 

JEG: [briefly engages in teohnology reoalibration] So let’s see if these help us. 

[...] See if you oan make the soreen green. 

Asa: Well... So let’s try to find green again. [A/K manipulate devioes] There. 

Stop! Yeah. 1 thi nk my theory’s right. Oh wait, but we both have to be 
moving, ‘oause when we stop... 

JEG: You both have to be moving? What do you. . .What do you mean by that? 

Asa: Seems so... Oh wait, no, that oan’t be possible, [to Kaylen] Stop! [soreen 

remains green even though hands are stationary] Yeah it’s not true. Go up. 

Asa temporarily oonsidered a new “green” theory — the hands must move oontinuously — 
but then immediately refuted this short-lived theory in light of empirioal data. 

JEG: You think you oan find another green [gestures to top half of the monitor] 

kind of up here...? 

Asa: My theory is obviously wrong. 

JEG: What was your theory before? 




11 



Asa: My theory was that there was a speeific [RH thumb/index, as before, 

gestures a vertieal interval] height that they have to be from eaeh other but 
[waves hand] that’s wrong [shakes head]. 

JFG: So what do you. . . What do you think now? 

Asa: Uhhm, 1 think that the height slowly inereases. [Kaylen concurs] 

In contrast to alternating their gazes between the hands and screen prior to the 
introduction of the crosshairs, when the crosshairs first appeared on the screen the 
children instantaneously gazed at them and thereafter almost never looked at their 
handheld devices while operating them. This transition from here to there apparently 
enabled Asa to re-examine his previous fixed-distance conjecture; in turn, the greater 
precision afforded by this transition to the screen enabled Asa to experience empirically 
information that he then interpreted as rendering his conjecture “obviously wrong.” 
Furthermore, the gestural-symbolic interfacing revealed one of Asa’s implicit framings; 
namely, that the devices/crosshairs “both have to be moving” so as to effect a green 
screen. At the point in the interview when the crosshairs were introduced, Asa had 
interacted with the artifact for approximately ten minutes with only modest progress 
along the dyad’s inquiry into the mystery device. Then, over a very brief period of time 
and without any expert modeling, Asa was able not only to disprove two of his 
conjectures (fixed difference, moving) but to produce a new conjecture, by which the 
distance (“height”) between the crosshairs should increase as they move up. In sum, Asa 
rapidly and successfully forded/blended the gestural-symbolic divide and immediately 
availed of supplemental inquiry affordances in this virtually extended problem space. 

Next, we discuss the case of Boaz, a 5*-grade male student indicated by his teachers as 
low achieving. The text below is a transcription from the early part of the interview, 
before the crosshairs had been revealed on the display. Boaz had been attempting to make 
the screen green, and DR asked Boaz to explain his green-making strategy. In his 
response, Boaz will be referring to a certain “camera.” This is not the video camera 
recording the session, but a sensor mounted on a tripod, part of the remote-action 
technology that picks up the infrared signal in the LED beam emitted by the handheld 
device the child operates. As we shall see, Boaz’s fixation on this amathematical device 
in his visual field impeded his incorporation of the virtual objects on the computer screen 
and thus delimited the scope and efficacy of his inquiry into the mystery device. 

DR: Well, then, how did you make it green? [gestures toward computer 

displays] 

Boaz: Ohl By... uhhm... 

DR: Are you figuring the green? 

Boaz: By, uhhhm... [swerves body to the left, orienting torso away from the 

screen and toward the sensor; points toward sensor with both palms] By. . . 
So, 1 guess these two things right here [points to devices] are showing the, 
uhmm [points to sensor] camera. Like [points back at devices] they’re 
looking at the camera, and, 1 guess, those light things on it [refers to an 
obsolete LED array that is still mounted to the sensor, a technological relic 
of an earlier MIT design] each have a different color in each column. And 
so when they move it up or down or equal [mimes accordingly], each one, 
like, I’m, it’s... it’s pinpointing.... 
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DR: So what would you do to make it green? 

Boaz: 1 would... [operating in the lower range of the embodied interaetion spaee, 
just above the desk, Boaz lifts and lowers the two deviees independently, 
monitoring for eolor on the large screen; he stops when the screen turns 
green and holds his hands fixed at that position] try pinpointing it by 
moving them up and down ‘til 1 made green. 

DR: Ok. So. . . [DR takes a mimed cue from DA] What if you tried to lift your 

hands up like this [lifts both hands in parallel into the upper ranges of the 
interaction space] and keep it green? [Boaz imitates by lifting his hands up 
in parallel above his shoulder height] Do you know any way to do that? 

Boaz: Oh! It would go out of the motion [gestures toward DR’s laptop]. It would 
go out of the camera [gestures toward sensor]. 

In the course of explaining to Boaz the boundaries of the physical interaction space, DR 
inadvertently mimes parallel hand motions, which would not effect a green screen. DR 
intends for these actions to inscribe for Boaz the interaction space in general. Boaz, 
however, does not share DR’s pragmatics intentions, and so he cannot differentiate 
between what DR takes to be technical and mathematical dimensions of the mimed 
action. Thus, when Boaz literally imitates the parallel motions that result in a red screen, 
he concludes that the approach must be strategically in error. 

When the crosshairs were eventually revealed, albeit he began tracking them in his 
peripheral vision, Boaz was still physically and perceptually oriented toward the sensor. 
As his alternating “1” and “it” utterances suggest, his attention wavered between his 
hands and their projection, between a gesture and its inscription, and thus between the 
technical and mathematical aspects of the situation. In contrast to Asa, Boaz did not 
objectify the vertical interval between the two devices as the thing to be controlled prior 
to the introduction of the crosshairs — his attention was still fixated on his hands rather 
than their relational property. Consequently, the crosshairs appeared to Boaz as 
something supplementary to the central activity itself, apparently complicating the 
interaction; in contrast, for Asa they afforded a means of evaluating existing theories. 

In sum, Boaz, unlike Asa, remained “stuck” between media, and his performance and 
understanding of their embedded mathematics suffered. Indeed, Boaz required direct 
assistance from the interviewers in “untangling” the para-medium and in-medium 
dimensions of the MIT. His difficulty in fording from gestural to symbolic media is not 
too unlike that of children in a Vygotsky’s study, who “when asked to draw good 
weather. . . .indicate the bottom of the page by making a horizontal motion of the hand, 
explaining, ‘This is the earth,’ and then, after a number of confused upward hatchwise 
motions, ‘And this is good weather’” (Vygotsky, 1930/1978, p. 108). To these children, 
as to Boaz, the focus remains on the gesture rather than on its projected inscription. 

Both the case of Boaz, in our study, and the child in Vygotsky’s study illuminate 
developmental challenges in negotiating media divides. Namely, these cases collectively 
frame sociocultural processes of acculturating novices into representational practices as 
supporting learners’ blending or hybridizing of embodied-gestural and virtual- 
inscriptional interaction orientations. Whereas this hybridity is crucial to cognitive- 
discursive empowerment and, thus, to valued participation in collective practices, the 
divide between the embodied and the symbolic may at times prove too great. 




13 



As designers, the heuristic image we sometimes invoke is of Janus, the mythological two- 
headed watcher of gateways and corridors. We find this image powerful, because it 
reminds us of the need to evaluate the artifacts we design and the learning corridors they 
provide from the perspectives both of an expert-designer and a novice-user. Through 
iterated design cycles, we strive to improve these gateways by creating technologies and 
protocols that afford invisible gestural-symbolical interaction blends. 

3.2 From Local Perceptuomotor Coupling to Global Schemes 

Naama is a Grade 5 female student indicated by her teachers as low achieving. In this 
section, we describe Naama’ s early attempts first to move her hands while keeping the 
screen green and then to articulate a stable rule for doing so. Ultimately, Naama was 
successful in determining the MIT’s multiplicative rules, yet we interpret her behaviors 
along the way as illuminating the complexity of grounding mathematical concepts in El 
activities. In particular, on the one hand we demonstrate how perceptuomotor fluency 
may form the basis for schematic reorganization, yet on the other hand we highlight 
possible difficulties in facilitating this process. 

Once Naama had first succeeded in positioning the crosshairs so as to make the screen 
green, the interviewers encouraged her to try moving her hands while still keeping the 
screen green. In her initial attempts to do so, Naama would first move both hands at a 
fixed distance from each other, which resulted in a red screen, and then she would correct 
to green either by returning both hands back to the previous green position or by 
adjusting the distance between her hands. Naama was thus able to inch her way up and 
down along the display by progressing from each green position to the next one, 
alternating between fixed-distance simultaneous bimanual actions yielding red followed 
by mono-manual sequential adjustment actions yielding green. We interpret the fixed- 
distance gestures as expressing Naama’ s theory-in-action. Because the theory was robust, 
Naama interpreted the “error” feedback (red screen) as indicating not a problem with her 
theory but production imprecision that does not bear on the theory-in-action but only on 
her perceptuomotor acuity and dexterity. That is, Naama tended implicitly to cast her 
adjustment manipulations as pragmatic “noise” irrelevant to the epistemic “signal” 
informing the evaluation of her theory. 

Soon after, however, Naama appears to have noticed new properties of the situation — she 
accordingly reverted to a new strategy, which she explained thus: 

The higher you go [right hand/crosshair], it [left hand/crosshair] has to follow, 
kind of. . . . If you want it [hands/crosshairs] to go higher, this one goes higher 
[right hand jolts twice upward] and then you have to move this one a little bit each 
time you move [left hand jolt once upward]. 

Interestingly, whereas Naama does not verbalize the quantitative properties inherent to 
the “two” vs. “one” upward jolts she gestured, these two-vs. -one jolts anticipated the 2:1 
ratio setting of the device. Indeed, this right-goes-up-two-units-then-left-goes-up-one-unit 
sequential strategy proved quite successful in terms of maintaining a green screen, 
barring the brief red interims. Naama was thus shifting from incorrect to correct theory- 
in-action — from: (a) simultaneous bimanual action maintaining fixed, unverbalized 
distance between the hands; to (b) sequential right-then-left quantified hand motions. 

This strategic shift indicates progress along a learning trajectory toward articulating an a- 
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per-/) proto-ratio conceptual structure (e.g., “two per one”). 

Then, asked again to explain her strategy, Naama reverted to the naive theory-in-action: 

You have to keep moving the hands and keep them in the same position. . . .kind of 

hold them in the same place. . . .you don’t move your hands out of the position. 

Executing this naive strategy, however, again required the performance of adjustment 
actions. But here we witness dissociation between the mathematical and performance 
value of strategies, a dissociation that we view as offering challenges for El design: 
Whereas Naama’ s theory was mathematically naive, and we were keen for her to switch 
to the other strategy that we appreciated as mathematically more advanced, Naama was 
eager to pursue the naive strategy, because she was becoming increasingly dexterous at 
performing perceptuomotor cycles, each consisting of a tiny fixed-distance jolt followed 
by a tiny adjustment. Naama appeared to prefer the naive strategy — it was more practical. 

One of the interviewers then asked Naama to lay down the tracking devises upon the desk 
and demonstrate her strategy using her bare hands. Note that by gesturing sans devices, 
Naama cannot receive online automated feedback on her gestures. One might therefore 
expect that Naama would move her hands at a fixed distance in accord with her explicit 
strategy and not perform any secondary adjustment actions, because there would be 
nothing to adjust by or to. However, and to our great surprise, whereas Naama announced 
she were moving her hands at a fixed distance, she simultaneously raised her hands 
miming a changing-distance action that approximated a correct 2: 1 growth! We asked her 
to repeat the demonstration, and she did so twice, still insisting that her hands were at a 
fixed distance. What are we to make of this acute gesture-speech mismatch? It could 
indicate that Naama knew more than she could as yet say; that she was prepared for 
conceptual change (see Alibali, Elevares, & Goldin-Meadow, 1997; Church & Goldin- 
Meadow, 1986). That is, it appears that the El activity entrained Naama to embody a 
pedagogically targeted action pattern before she could articulate it. 

The desired outcome of El conceptual learning activities is of course not a physical 
performance per se, but the development of an articulated scheme governing relations 
between key properties of this performance. In particular, our design was for participants 
to verbalize initially the relation between the hands’ height and distance (“the higher you 
go, the bigger the distance”) and then between the two hands’ respective increments (“for 
every two we go up on this side, we go up one on this side”). In accord with this design 
objective, once the grid and numerals had been introduced onto the display, Naama orally 
likened the upward vertical gestures to “stacking blocks,” two blocks on the right for 
every one block on the left. Whereas this verbal assertion echoed her earlier two-per-one 
gestured utterance, it arrived only after enduring manifestations of the fixed-distance 
strategy. Ironically, introducing the grid and numerals into the problem space — 
intervention actions that ostensibly should have empowered Naama — may have impeded 
her reflective abstraction on the budding “higher-bigger” embodied insight. Moreover, 
this premature instrumentalization of Naama’s working space may have strengthened her 
fixed-distance strategy, because these mathematical instruments provided her with a 
familiar context to verbalize her naive strategy arithmetically. In sum, whereas EI- 
induced perceptuomotor competence is necessary, it is not sufficient: embodied theory- 
in-action that remains uncultivated may never contribute to mathematical understanding. 
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because learners may revert to safe, workable “detour” solutions that do not navigate 
unfamiliar territory and are thus less cognitively challenging. 

Currently, we can only conjecture a microgenetic account of how learners operating the 
MIT shift from fixed-distance to changing-distance theory-in-action. Namely, we do not 
know how, if at all, they reinterpret their adjustment actions as challenging their fixed- 
difference theory-in-action. We can only speculate that the learners become cognizant of 
consistency in the direction of the corrective adjustments: as their hands ascend, they 
notice that effective adjustments are always those that make the distance larger, not 
smaller. An alternative yet not mutually exclusive speculation is as follows. As learners 
become more proficient in engaging the tight local perceptuomotor loops and, thus, 
perform the overall gestures faster as well as draw less on cognitive resources, the overall 
figural pattern becomes more salient to them, and this global pattern then obtains to the 
local loops, which in turn become more efficient, and so on recursively, until the 
changing-distance action becomes too compelling to misinterpret. And yet, as we have 
witnessed, a robust naive theory can go a long way in blinding learners to the “truth”. . . 

More broadly, the discrepancy between Naama’s embodied and articulated knowledge — 
between what Naama “did” and “said” — bears on our understanding of the 
technologically entrained perceptuomotor loops (action-feedback-adjustment) and in 
particular how these differ from a pedagogically targeted scheme yet contribute to its 
construction. For the child to endorse adjustment actions as epistemically inherent to a 
new emerging scheme rather than as pragmatically corrective of execution incompetence, 
the child has to disengage from the sequence of micro-perceptuomotor loops and engage 
in reflective abstraction on this activity (Karmiloff-Smith, 1992; Piaget & Inhelder, 

1969) — the child has to switch from the local “trees” and see the global “forest.” Yet, 
paradoxically, in doing so the child has recourse only to existing schemes as resources for 
building a new theory (Karmiloff-Smith & Inhelder, 1978). The child will therefore hold 
on to this theory until empirical counter-evidence becomes so salient it compels 
reevaluating the theory (Karmiloff-Smith, 1988). The new theory is initially articulated as 
a variant on, or in contradistinction to the existing theory, so that multiplicative reasoning 
could emerge as qualifying additive reasoning. It is thus that a collection of infinitesimal 
micro-perceptuomotor loops may become “ironed out” and integrated globally in 
schematic redescription vis-a-vis an emerging empirical correlation.^ 

Finally, as instructional designers we are stuck by phenomenological similarity of direct 
control of concrete instructional artifacts (“manipulatives”) and remote control of virtual 
objects; by these actions’ similar exploratory and discursive pedagogical affordances. 
Moreover, bimanual gesture itself might be viewed as an object-to-think-with — an 
immaterial action-object for sure, yet a thing that nevertheless can be reproduced, 
refined, referred to, analyzed, quantified, elaborated, debated, negotiated, and signified. 
That said, Naama’s case is testimony to cognitive and interactional challenges inherent to 



^ Inquiring into the epistemological paradox of perceiving unfamiliar structure in the absence of explicit 
guidance, C. S. Peirce wrote of “hypostatic abstraction” (Hoffmann, 2003). Our study participants had not 
only to perceive new structure but do so against the backdrop of a functioning structure (the naive theory). 
Yet our participants did not work alone but rather with an instructor who tactically and pragmatically 
manipulated their actions so as to draw their attention to conceptually relevant properties in the perceptual 
field and thus render an alternative structure plausible. 
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helping students re-see gestured “objects” in light of new conceptual structure. 

3.3 Hooks and Shifts: Reappropriating Symbolic Art facts 

The dual construct of “hook and shift” was developed during the current project in an 
attempt to characterize a surprising behavioral pattern of students who were observed to 
spontaneously bootstrap mathematical forms through appropriating artifacts in the 
process of problem-solving a challenging task (Abrahamson, Trninic, Gutierrez, Huth, & 
Lee, under review). A hook is a subjective and contextual affordance of a mathematical 
artifact that students recognize as they become cognizant of the artifact’s availability in 
the course of solving a problem and communicating their solution. Specifically, students 
engage the artifact as an enactive, explanatory, or evaluative utility — as a way of better 
implementing their practical competence or as a semiotic means of objectifying their 
unarticulated strategy. A shift is the students’ consequent unanticipated, unpremeditated, 
undemonstrated, yet more mathematically sophisticated view of the problem situation 
that emerges through using the artifact (see Schon, 1992, on "see. ..move.. .see"). We have 
identified a set of interaction parameters that may enable or hinder hooks and shifts in 
artifact-mediated discovery-based learning (Abrahamson et ah, under review). The 
current section demonstrates an occurrence of hook-and-shift. Albeit, at this time is it not 
clear how one might view this pattern as unique to El design. Whereas the primary 
objective of this section is to enrich and ground our definitions of hook-and-shift, the 
dyadic interaction of the specific data episode selected for this section offers a 
supplementary, if complicating, dimension. Yet we believe this added complexity is 
ultimately rewarding, because its unique discursive qualities bring out in relief the very 
processes that we wish to demonstrate and understand. 

Eden and Uri, two Grade 6 male participants, were selected for a paired interview on the 
basis of compatible mathematical achievement (both were identified by their teachers as 
high achievers). Their interview was conducted by an apprentice researcher (DT), with 
the lead researcher (DA) occasionally intervening. Eden and Uri were seated side by side 
in front of the MIT remote-action sensor system and computer display and each operated 
one of the two tracker devices (right- tracker device [RT] and left- tracker device [ET]). 
The students were presented with the task of making the screen green under an unknown 
1:2 ratio setting (i.e., the screen would be green only when the right-crosshair [Re] were 
double as high as the left-crosshair [Lc]). 

3.3.1 Hooking to the grid. Prior to the introduction of the grid, Eden and Uri had been 
working together for nearly 1 1 minutes in the no-crosshairs (blank screen) condition and 
then another 7 minutes in the crosshairs condition. So doing, they identified two spatial 
dimensions — height and distance — as relevant to making the screen green and had 
articulated two theorems with regard to each of these dimensions: (a) Re should be higher 
than Ec; and (b) the vertical distance between Re and Ec is non-arbitrary. However, Eden 
and Uri disagreed as to whether this vertical distance should change or remain constant as 
the crosshairs move. Whereas both Uri and Eden observed different distances between 
the Re and Ec at certain green locations, Uri interpreted this difference as a systemic 
principle for making green, while Eden attributed it to an HCI issue, as though the 
physical manipulation were inaccurate (Eden, apparently an avid video-game designer, 
referred to this error as the “human factor”). Uri articulated a covariant principle relating 
distance and height, explaining that “it has to get, like, farther away, the higher up we 
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are” and that “the lower you are, the less distanee apart it has to be” — a changing- 
distanee theorem- in- aetion. Eden, however, eourteously responded with, “Well I’m not 
sure if it matters if you’re lower or higher, but 1 think it’s just, like, you stay the same 
distance apart” — a yixeJ-distance theorem-in-action. 

Thus, Uri and Eden’s collaborative hands-on problem solving enabled them each to 
notice and explicitly articulate a relation between the crosshairs’ height and distance, yet 
whereas Uri concluded from their empirical data that the distance should vary, Eden 
concluded from the same data that it should not. This disagreement bore practical 
implications, because the dyad were co-operating the two devices — they each depended 
on the other to enact a green-making theorem-in-action, yet their respective theorems 
were mutually exclusive. Consequently, the students’ success within this collaboration 
became contingent on whether or not they could rule between their incompatible fixed- 
distance and changing-distance theorems-in-action. At the same time, they apparently felt 
under-equipped to arbitrate in the continuous space. Namely, when the grid was 
subsequently introduced (see below), they recognized its potential for ruling between the 
theorems — they “hooked” to the grid largely for its discursive, argumentation, and 
arbitration affordances. Specifically, the grid served these boys to quantify the distance 
between the crosshairs and ultimately determine that this distance should in fact change 
between green spots, as Uri had believed. Eden soon concurred. The excerpt below 
begins immediately after DT had layered the grid onto the screen.^ 

Eden: Grid. 

Uri: Yeah. [Grabs RT, lifts it, and remote-places Re on the U‘-from-the-base- 

line gridline (hence “Re up to 1-line”). Simultaneously, Eden, too, brings 
Ec up to 1-line. On the way up, between 0-line and 1-line, the screen 
flashes green for a moment but then turns red. Eden lowers Ec back down, 
holds it at .5 units. The screen turns green.] Oh so you can like show 
where... Eet’s see, so [Re up from 1-line to 2-line]// if you’re on here... 

Eden: //maybe it has to be two. . . [Ec up to 1-line (see Eigure 4a, next page)] an 
entire box apart. 

Uri: [Re up to 3-line] If 1 go here. . . 

Eden: [Ec up to 2-line; screen goes red (see Eigure 4b, next page)] Then maybe 
you should raise it [Uri raises Re to just below 4-line; screen flashes 
green]. So maybe the higher you go, the more boxes it is apart. 

Uri: Eet’s just say like I’m here [Re down to 2-line], then he has to be one box 

under me... 

Eden: [Ec down to 1-line; screen goes green] And then the higher he goes// 

Uri: //and when 1 go here [Re up to 3-line], he has to be like in the middle 

[Eden moves Ec up to 1.5 units; screen goes green] 

Eden: So the higher// 

Uri: //And here [Re up to 4-line, while Eden moves Ec up to 2-line] he has to 

be like two boxes under me. 

Eden: So like the higher it goes, the more space there has to be between each. 
[Both Eden and Uri place their tracker devices on the table] 



^ RT = Right-Tracker device; LT = Left-Tracker device; Rc = Right crosshair; 
Lc = Left crosshair; // = overlap by next speaker. 
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Figure 4b. Immediately, Uri and Eden 
reposition Re and Lc to 3-line and 2-line, 
respeetively. The sereen turns red. Upon 
noticing that the fixed-distance theory does 
not obtain, Eden says, “Then maybe you 
should raise it. So maybe the higher you go, 
the more boxes it is apart.” 



Figure 4a. After the introduction of the grid, 
Uri (middle) and Eden (far right) find green 
with Re at 2-line and Ec at 1-line, 
respectively. Noticing the distance between 
the Re and Lc, Eden predicts that the fixed- 
distance subtends “an entire box.” 



Thus, it appears that both Eden and Uri immediately appropriated the grid in view of its 
affordances to arbitrate among their conflicted theorems, however they differed with 
respect to the nature of their discovery, and this difference can be related to their 
idiosyncratic beliefs prior to the introduction of the grid. Namely, Uri had articulated a 
changing covariant relation between height and distance, so for him the grid afforded 
reiterating and quantifying this qualitative principle. Specifically, the grid enabled Uri to 
reformulate his continuous qualifier “get farther away” as the discrete quantifiers “one 
box” and then “two boxes.” Eden, who had acknowledged the in-principle possibility of a 
changing-distance rule yet nevertheless maintained a fixed-distance rule, soon changed 
his mind and articulated a changing-distance hypothesis (“the higher you go, the more 
boxes it is apart”). However, Eden ends with a qualitative statement about “space,” which 
suggests that he construed the grid as a means not of quantifying the “higher-bigger” 
conjecture but of evaluating whether or not this conjecture even obtained. Thus, Eden and 
Uri both hooked to the same artifact, yet they utilized it for different purposes. This 
episode demonstrates the plausible view that an artifact’s subjective utility is contingent 
on the individual’s goals. The episode also suggests that a dyad can engage in 
collaborative hands-on problem solving even as they hold different theorems-in-action 
(cf Sebanz & Knoblich, 2009) 

In the following excerpt, below, we will continue at a point where the dyad initiated 
further inquiry. As we shall see, the dyad’s exploration will shift them from the now- 
consensual “higher-bigger” strategy to a proto-ratio a-per-h strategy. Both strategies can 
be viewed as expressing covariation — “the more x, the morey.” However, the former 
strategy is continuous-qualitative, whereas the latter is discrete-quantitative, so that 
student adoption of the latter strategy is a pedagogically desirable outcome. 

3.3.2 Shifting with the grid. Having reached consensus, Eden and Uri elaborate their 
explanation. They are now instrumented with the grid’s quantification affordances, and 
this new conceptualization of space will engender the semi-spontaneous emergence of a 
new mathematical form. In particular, in the transcription that follows we will observe 
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that the students shift with the grid from the continuous-qualitative strategy to a discrete- 
quantitative strategy. 

Uri: I think, like, uhm, when I go up to here [points to 2-line], he has to be one. 

Then when I go up// 

Eden: Like for every. . . for every box he goes up, I have to move, go downV 

Uri: //You have to go up half// //a box. 

Eden: //Y eah// 

The dyad’s coordinated production of green tacitly modulated from simultaneous 
motions, in which the distance constantly increases and green coloration is maintained 
throughout, to sequential motions, in which each hand separately ratchets up to its 
respective designated destination and green is effected after a brief red interim, once the 
second ratcheted motion is completed. Imperceptibly, the linguistic constraints of the 
speech modality as well as the linearity of discursive turn-taking and ownership over 
actions thus shifted the dyad from their higher-bigger continuous-qualitative strategy to 
an a-per-h discrete-quantitative proto-ratio strategy. 

Eden and Uri thus co-discovered that in order to maintain green, they should progress at 
intervals of 14 (Eden) and 1 (Uri) coordinated vertical intervals, either both going up the 
screen or both going down. It is through this serendipitous discovery that their earlier 
observation, “The higher you go, the more boxes it is apart,” a covariation between 
height and distance, transformed (shifted) into a covariation that foregrounds the 
independent actions of the left and right crosshairs, “Eor every box he goes up — you have 
to go up half,” a new strategy that is closer to normative forms for ratio (i.e., a-per-h). We 
wish to underscore that whereas the general x-per-y covariation form was maintained, its 
semantic-mathematical content was replaced (see Eigure 5, below). 



Mathematical Properties “The more x, ...” “the more y.” 

Continuous-qualitative: “The higher you go, ... ” “the bigger the distance” 

Discrete-qualitative: “The higher you go, ... ” “the more boxes it is apart” 

Discrete-quantitative: “Eor every box he goes up, . . . ” “you have to go up half’ 

Figure 5. “Covariation” linguistic structure across strategy shifts. 

In addition to explicating our hook-and-shift construct, our analysis of the case has 
demonstrated that collaborative mathematical learning processes are impacted by nuances 
of personal/interpersonal framing to the extent of dissociation between a dyad’s physical 
and epistemic actions. Namely, whereas the two dyad members collaborated on using a 
single symbolic artifact (the grid), their joint experiment simultaneously enacted an 
exploration of two different hypotheses (“same distance” and “different distance”). Uri 
was quite comfortable from the very onset with the higher-bigger principle, so he did not 



On several occasions throughout the interview, Eden used “up” to refer to the direction we would call 
“down” and vice versa. As it turned out, Eden is an expert video gamer. He thus may have been using 
schemes that are natural to video-gaming culture. Therein, and counter to non-gamer discourse, joystick 
actions are often mapped onto virtual actions such that swinging the joystick up-forward results in the first- 
person avatar looking or moving down. 
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need arbitration but refinement, whereas Eden, who had challenged Uri with a fixed- 
distance theorem, needed resolution. Once a common ground was established, the dyad 
was able to continue mathematizing the mystery artifact-phenomenon and jointly 
articulate a new mathematical form, which we recognized as pedagogically desirable. 



4. Discussion and Conclusion 

The hybrid or the meeting of two media is a moment of truth and 
revelation from which new form is born. . ..The moment of the meeting of 
media is a moment of freedom and release from the ordinary trance and 
numbness imposed.... on our senses. (McLuhan, 1964, p. 63). 

Even as learning scientists are increasingly accepting a view of mathematical reasoning 
as multimodal spatial-temporal activity, technological advances and free-market forces 
bode an impending ubiquity of personal devices capable of utilizing remote-embodied 
input. Poised between theory and industry, design-based researchers are only beginning 
to wrap their minds around the protean marriage of embodied cognition and remote 
action. Embodied interaction, a form of physically immersive instrumented activity, is 
geared to augment everyday perceptuomotor learning by fostering cognitive structures 
that leverage homo sapiens’ evolved capacity to orient and navigate in a three- 
dimensional space, wherein the brain developed by and for action (Dourish, 2001). 

We currently have far more questions than answers respecting the prospects of El as well 
as principles for best design and facilitation of these innovations. Our strategy has been to 
engage in conjecture-driven cycles of building, testing, and reflecting. In this spirit, the 
current paper aimed to share our excitement with El and offer some early observations 
and caveats. Yet, as often occurs when new media are encountered, we learn as much 
about pan-mQ&ia. practice as about the new medium per se. As such, the “ontological 
innovations” we have stumbled upon through our design-based research appear to bear 
more generally on how people do and could learn (cf diSessa & Cobb, 2004). 

4.1 El Learning Affordances 

Our analyses depicted El learning as the evolution of users’ subjective meaning for 
automated sensory stimuli and, in particular, the role of perceptuomotor skill as a vehicle 
or platform for the mediation of conceptual development. El automated information (such 
as “green” in our design) evolved in the functional and cognitive roles it played. That is, 
green: (a) began as the task objective', (b) soon became the perceptuomotor feedback, as 
the users attempted to complete the task objective; and (c) came to hold together a set of 
otherwise unrelated hand-location pairs sharing a common effect and begging a name. 
Eeedback on perceptuomotor performance thus came to demarcate the set of “green” 
number pairs as all belonging to an emerging ontology. In this sense, green functioned as 
more than an objective or feedback — it served as an ontological scaffold or conceptual 
placeholder. Ultimately, the scaffold collapses, or the placeholder is filled, once users 
determine the activity’s mathematical rule and recognize the rule’s power for 
anticipating, recording, and communicating the MIT’s solution procedure. As one child 
gleefully quipped, upon determining the multiplicative relation of an unknown ratio, “I 
hacked the system!” 
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El creates arenas for body-based mediation of qualitative mathematieal notions. As the 
students engage in problem-solving our MIT mystery deviee, their physieal actions 
inscribe a ehoreographed form with inereasing deftness. Whereas the student views these 
forms as physieal solution proeedures, the edueator — ^who views the forms from the 
vantage point of an expert’s diseiplinary perspective — eoneeptualizes these forms as 
multimodal image sehema underlying the eognition of the targeted eoneept. Using 
representational resourees and diseursive guidanee, the instruetor may then steer students 
to progressively signify these image sehema into what beeome eoneeptual metaphors of 
the emerging mathematieal ideas. That is, even as the gestured forms lend meaning to 
mathematical propositions, they take on the epistemological role of metaphorieal 
simulations (oonoept-speoifie “math kata,” if you will). In praetiee, the mathematieal 
eoneept emerges when students utilize new mathematieal symbolie artifaets, whieh the 
instructor introduces into the problem-solving spaee, as means of enaeting, enhaneing, 
explaining, and/or evaluating their solution proeedure. As such, mathematical knowledge 
emerges through reeognizing a partieular eultural form (e.g., the Cartesian grid) as 
eontextually useful equipment (Radford, 2003). Speeifieally, students utilize these 
available forms to articulate their physieal solution proeedures, first multimodally 
(verbally and gesturally) and then also symbolically (by utilizing numerical inscriptions). 
Initially, these artieulations are naive and qualitative, but they progressively adhere to 
mathematieal forms through situated aseension from physieal to mathematieal. 

In partieular, we have been struek by learners’ eapaeity to strategieally re-appropriate 
available mathematieal instruments in ways that have not been demonstrated for them yet 
bootstrap their eoneeptual praetiee. We have named this behavioral pattern “hook and 
shift,” beeause the learners first adopt the artifaets to serve the enaetion, enhaneement, 
explanation, or evaluation of a eurrent strategy yet through doing so they stumble upon 
the artifaets’ embedded affordanees that beeome revealed only through engaging them 
purposefully, so that the students reeonfigure their strategy is ways that eo-opt these 
powerful affordanees. For example, note the ease presented above where students, by 
engaging the grid, shifted from a “the higher, the bigger” strategy to the pedagogieally 
advantageous proto-ratio and multiplieative strategies. 

4.2 Toward a Heuristic Design Framework for Mathematical El 

Emerging from our researeh are the following empirically grounded prineiples of a 
heuristie design framework for El mathematics problem-solving learning aetivities, that 
we ean only sketeh here: (1) The designer seleets/engineers a learning environment that 
ineludes a deviee linking simple eoordinated aetions remotely to generic virtual objects; 
(2) The designer plans and implements progressive-mathematization supports in the form 
of layerable/removable symbolie artifaets; (3) Students’ physieal aetion should not only 
enable the gathering of data but aetually eonstitute an integral eomponent of the data; (4) 
Moreover, students’ physieal solution proeedure has to inscribe an image sehema of the 
targeted mathematieal notion; (5) The inquiry should be self-adaptive, not preseriptive, so 
that eaeh child can gather the data they need when they need it; (6) The student should be 
able to move baek and forth between embodied and symbolieal eontrol operations; and 
(7) The student should be supported in eoordinating various meanings emerging from the 
activity by explicating relations among different strategies they discover. 
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Learning trajectories following this plan will likely be supported by, or fraught with 
interaction features exemplified by our study. Namely: (a) Students may encounter 
difficulty in homing on the core mathematical issues, if the El technology is too obdurate, 
obtrusive, or otherwise unergonomic; (b) Students are liable to become competent at an 
El task yet not know what they know, that is, they may perform well yet frame their 
performance by incompatible theory (Karmiloff-Smith, 1988), such that the pedagogical 
utility of their competence may remain untapped unless ushered forth; and (c) Students 
engaged in codependent El tasks need to communicate effectively and work flexibly 
during their explorations, because they are liable to bear conflicting theories-in-action 
that, in turn, could impede the coproduction of targeted enactment objectives. 

In sum, we view El as bearing the capacity of supporting transformative teaching and 
learning. Specifically, El enhances the implementation of visionary design frameworks, 
by which students should begin inquiry into complex mathematical concepts through 
presymbolic action-based reasoning (Eorman, 1988; Nemirovsky, 2003; Thompson, 

1993). More generally, we submit, El activities constitute rewarding empirical contexts 
for research aiming to deepen and expand our field’s understanding of how instructors 
discipline learners’ perception of a shared domain of scrutiny (Stevens & Hall, 1998). 

Whereas our work is in its early stages and our conclusions tentative, we hope to have 
conveyed enthusiasm over El’s unique instructional and theoretical affordances. Our 
future work will continue to seek improvements in both theory and design, availing of 
recent hands-free El development. In this spirit, we offer the following lagniappe. 

4.3 El as a Transformative Learning Medium 

New technological forms for enacting cultural practice bear the capacity to 
“restructurate” a domain so as to increase fluency, deepen understanding, and 
ultimately — as production prices plummet — democratize access (Blikstein & Wilensky, 
2009; Brock & Price, 1980; Wilensky & Papert, 2010). In particular, even more so than 
“hands on” microworlds (e.g., Edwards, 1995; Kaput & West, 1994; Schwartz & 
Yerushalmy, 1993), El-enabled “hands m” perceptuomotor exploration is ideally 
unencumbered by conceptually peripheral constraints of manipulation and typography 
inherent to traditional media and inscriptional systems. Thus, as the body settles into its 
skillful invisibility (Dreyfus, 1990), more cognitive resources should become available 
for tinkering creatively in the embodied microworld, by which the body instruments itself 
into a disciplinary domain of practice. 

Yet restructurations of content may bear implications for restructuring theoretical models 
pertaining to this and other content learning. Specifically, whereas the hook-and-shift 
construct appears to agree with views of learning as discovery (Ereudenthal, 1986) or 
creativity (Bamberger & Schon, 1983), our evidence of wnmediated creative discovery 
complements sociocultural conceptualizations of learning as imitation, apprenticeship, or 
legitimate peripheral participation (Eave & Wenger, 1991; Eee & Majors, 2000; Rogoff, 
1990). We thus anticipate that El design stands to play a pivotal role in creating 
productive arenas for “dialectical” empirical research that strives to draw on both 
cognitive and sociocultural theory so as to build coherent models of conceptual 
ontogenesis in the interactive milieu (cf diSessa, 2008). 




23 



Finally, the analytic orientation exemplified above suggests an intriguing line of research 
that we view as bearing great potential, namely conceptualizing how master mathematics 
teachers steer student engagement in activities centered on the mediated performance of 
physical actions. The ultimate goal of this line of research would be to develop an 
embodied-cognition model of grounded mathematics instruction. Yet this line of research 
could look to diverse cultural-historical forms of physical performance, such as music, 
dance, and the martial arts as research entries into pedagogical acumen. The skills 
inherent to these cultural practices might, at first blush, be viewed as aconceptual and, as 
such, hardly bearing on mathematical reasoning and learning. However, the recent 
theorization of mathematical cognition as grounded in multimodal, spatial-temporal 
imagistic simulations suggests that the phenomenology of mathematical reasoning draws 
on the same perceptuomotor resources as do human practices with manifest external 
physical production. From this perspective, mathematical concepts are articulated 
reifications of emergent embodied notions in conventional semiotic forms (see Radford, 
2003) whose cultural mediation is enabled by shared biology and available artifacts 
(Nunez, Edwards, & Matos, 1999). Moreover, an implication of simulated multimodal 
action as the implicit fodder of mathematical cognition should encourage us to look 
closer at the pedagogical role of non-propositional information in educational exchanges 
(e.g., Alibali, Flevares, & Goldin-Meadow, 1997). There, and not in rote procedures for 
symbol manipulation, may lie the heart of the matter that students and researchers alike 
have been groping for. 
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